The Sustainability of New Programs and Innovations: A Review of the Empirical Literature and Recommendations for Future Research

Background: The introduction of evidence-based programs and practices into healthcare settings has been the subject of an increasing amount of research in recent years. While a number of studies have examined initial implementation efforts, less research has been conducted to determine what happens beyond that point. There is increasing recognition that the extent to which new programs are sustained is influenced by many different factors and that more needs to be known about just what these factors are and how they interact. To understand the current state of the research literature on sustainability, our team took stock of what is currently known in this area and identified areas in which further research would be particularly helpful. This paper reviews the methods that have been used, the types of outcomes that have been measured and reported, findings from studies that reported long-term implementation outcomes, and factors that have been identified as potential influences on the sustained use of new practices, programs, or interventions. We conclude with recommendations and considerations for future research. Methods: Two coders identified 125 studies on sustainability that met eligibility criteria. An initial coding scheme was developed based on constructs identified in previous literature on implementation. Additional codes were generated deductively. Related constructs among factors were identified by consensus and collapsed under the general categories. Studies that described the extent to which programs or innovations were sustained were also categorized and summarized. Results: Although “sustainability” was the term most commonly used in the literature to refer to what happened after initial implementation, not all the studies that were reviewed actually presented working definitions of the term. Most study designs were retrospective and naturalistic. Approximately half of the studies relied on self-reports to assess sustainability or elements that influence sustainability. Approximately half employed quantitative methodologies, and the remainder employed qualitative or mixed methodologies. Few studies that investigated sustainability outcomes employed rigorous methods of evaluation (e.g., objective evaluation, judgement of implementation quality or fidelity). Among those that did, a small number reported full sustainment or high fidelity. Very little research has examined the extent, nature, or impact of adaptations to the interventions or programs once implemented. Influences on sustainability included organizational context, capacity, processes, and factors related to the new program or practice themselves. Conclusions: Clearer definitions and research that is guided by the conceptual literature on sustainability are critical to the development of the research in the area. Further efforts to characterize the phenomenon and the factors that influence it will enhance the quality of future research. Careful consideration must also be given to interactions among influences at multiple levels, as well as issues such as fidelity, modification, and changes in implementation over time. While prospective and experimental designs are needed, there is also an important role for qualitative research in efforts to understand the phenomenon, refine hypotheses, and develop strategies to promote sustainment

A randomized controlled dismantling trial of post-workshop consultation strategies to increase effectiveness and fidelity to an evidence-based psychotherapy for Posttraumatic stress disorder

Abstract: Background: Posttraumatic Stress Disorder (PTSD) is a serious mental health condition with substantial costs to individuals and society. Among military veterans, the lifetime prevalence of PTSD has been estimated to be as high as 20%. Numerous research studies have demonstrated that short-term cognitive-behavioral psychotherapies, such as Cognitive Processing Therapy (CPT), lead to substantial and sustained improvements in PTSD symptoms. Despite known benefits, only a minority of clinicians provide these therapies. Transferring this research knowledge into clinical settings remains one of the largest hurdles to improving the health of veterans with PTSD. Attending a workshop alone is insufficient to promote adequate knowledge transfer and sustained skill; however, relatively little research has been conducted to identify effective post-training support strategies. Methods: The current study investigates whether clinicians receiving post-workshop support (six-month duration) will deliver CPT with greater fidelity (i.e., psychotherapy adherence and competence) and have improved patient outcomes compared with clinicians receiving no formal post-workshop support. The study conditions are: technology-enhanced group tele-consultation; standard group tele-consultation; and fidelity assessment with no consultation. The primary outcome is independent assessment (via audio-recordings) of the clinicians’ adherence and competence in delivering CPT. The secondary outcome is observed changes in patient symptoms during and following treatment as a function of clinician fidelity. Post-consultation interviews with clinicians will help identify facilitators and barriers to psychotherapy skill acquisition. The study results will inform how best to implement and transfer evidence-based psychotherapy (e.g., CPT) to clinical settings to attain comparable outcomes to those observed in research settings. Discussion: Findings will deepen our understanding of how much and what type of support is needed following a workshop to help clinicians become proficient in delivering a new protocol. Several influences on clinician learning and patient outcomes will be discussed. An evidence-based model of clinical consultation will be developed, with the ultimate goal of informing policy and influencing best practice in clinical consultation. Trial registration: ClinicalTrials.gov: NCT0186176

Protocol for the ROSE sustainment (ROSES) study, a sequential multiple assignment randomized trial to determine the minimum necessary intervention to maintain a postpartum depression prevention program in prenatal clinics serving low-income women

Background: More research on sustainment of interventions is needed, especially return on investment (ROI) studies to determine cost-benefit trade-offs for effort required to sustain and how much is gained when effective programs are sustained. The ROSE sustainment (ROSES) study uses a sequential multiple assignment randomized (SMART) design to evaluate the effectiveness and cost-effectiveness of a stepwise approach to sustainment of the ROSE postpartum depression prevention program in 90 outpatient clinics providing prenatal care to pregnant women on public assistance. Postpartum depression (PPD) is common and can have lasting consequences. Outpatient clinics offering prenatal care are an opportune place to provide PPD prevention because most women visit while pregnant. The ROSE (Reach Out, Stay Strong, Essentials for mothers of newborns) program is a group educational intervention to prevent PPD, delivered during pregnancy. ROSE has been found to reduce cases of PPD in community prenatal settings serving low-income pregnant women. Methods: All 90 prenatal clinics will receive enhanced implementation as usual (EIAU; initial training + tools for sustainment). At the first time at which a clinic is determined to be at risk for failure to sustain (i.e., at 3, 6, 9, 12, and 15 months), that clinic will be randomized to receive either (1) no additional implementation support (i.e., EIAU only), or (2) low-intensity coaching and feedback (LICF). If clinics receiving LICF are still at risk at subsequent assessments, they will be randomized to either (1) EIAU + LICF only, or (2) high-intensity coaching and feedback (HICF). Additional follow-up interviews will occur at 18, 24, and 30 months, but no implementation intervention will occur after 18 months. Outcomes include (1) percent sustainment of core program elements at each time point, (2) health impact (PPD rates over time at each clinic) and reach, and (3) ROI (costs and cost-effectiveness) of each sustainment step. Hypothesized mechanisms include sustainment of capacity to deliver core elements and engagement/ownership. Discussion: This study is the first randomized trial evaluating the ROI of a stepped approach to sustainment, a critical unanswered question in implementation science. It will also advance knowledge of implementation mechanisms and clinical care for an at-risk population

Proceedings of the 3rd Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2015: advancing efficient methodologies through community partnerships and team science : Seattle, WA, USA. 24-26 September 2015.

Introduction to the 3rd Biennial Conference of the Society for Implementation Research Collaboration: advancing efficient methodologies through team science and community partnerships Cara Lewis, Doyanne Darnell, Suzanne Kerns, Maria Monroe-DeVita, Sara J. Landes, Aaron R. Lyon, Cameo Stanick, Shannon Dorsey, Jill Locke, Brigid Marriott, Ajeng Puspitasari, Caitlin Dorsey, Karin Hendricks, Andria Pierson, Phil Fizur, Katherine A. Comtois A1: A behavioral economic perspective on adoption, implementation, and sustainment of evidence-based interventions Lawrence A. Palinkas A2: Towards making scale up of evidence-based practices in child welfare systems more efficient and affordable Patricia Chamberlain A3: Mixed method examination of strategic leadership for evidence-based practice implementation Gregory A. Aarons, Amy E. Green, Mark. G. Ehrhart, Elise M. Trott, Cathleen E. Willging A4: Implementing practice change in Federally Qualified Health Centers: Learning from leaders’ experiences Maria E. Fernandez, Nicholas H. Woolf, Shuting (Lily) Liang, Natalia I. Heredia, Michelle Kegler, Betsy Risendal, Andrea Dwyer, Vicki Young, Dayna Campbell, Michelle Carvalho, Yvonne Kellar-Guenther A3: Mixed method examination of strategic leadership for evidence-based practice implementation Gregory A. Aarons, Amy E. Green, Mark. G. Ehrhart, Elise M. Trott, Cathleen E. Willging A4: Implementing practice change in Federally Qualified Health Centers: Learning from leaders’ experiences Maria E. Fernandez, Nicholas H. Woolf, Shuting (Lily) Liang, Natalia I. Heredia, Michelle Kegler, Betsy Risendal, Andrea Dwyer, Vicki Young, Dayna Campbell, Michelle Carvalho, Yvonne Kellar-Guenther A5: Efficient synthesis: Using qualitative comparative analysis and the Consolidated Framework for Implementation Research across diverse studies Laura J. Damschroder, Julie C. Lowery A6: Establishing a veterans engagement group to empower patients and inform Veterans Affairs (VA) health services research Sarah S. Ono, Kathleen F. Carlson, Erika K. Cottrell, Maya E. O’Neil, Travis L. Lovejoy A7: Building patient-practitioner partnerships in community oncology settings to implement behavioral interventions for anxious and depressed cancer survivors Joanna J. Arch, Jill L. Mitchell A8: Tailoring a Cognitive Behavioral Therapy implementation protocol using mixed methods, conjoint analysis, and implementation teams Cara C. Lewis, Brigid R. Marriott, Kelli Scott A9: Wraparound Structured Assessment and Review (WrapSTAR): An efficient, yet comprehensive approach to Wraparound implementation evaluation Jennifer Schurer Coldiron, Eric J. Bruns, Alyssa N. Hook A10: Improving the efficiency of standardized patient assessment of clinician fidelity: A comparison of automated actor-based and manual clinician-based ratings Benjamin C. Graham, Katelin Jordan A11: Measuring fidelity on the cheap Rochelle F. Hanson, Angela Moreland, Benjamin E. Saunders, Heidi S. Resnick A12: Leveraging routine clinical materials to assess fidelity to an evidence-based psychotherapy Shannon Wiltsey Stirman, Cassidy A. Gutner, Jennifer Gamarra, Dawne Vogt, Michael Suvak, Jennifer Schuster Wachen, Katherine Dondanville, Jeffrey S. Yarvis, Jim Mintz, Alan L. Peterson, Elisa V. Borah, Brett T. Litz, Alma Molino, Stacey Young McCaughanPatricia A. Resick A13: The video vignette survey: An efficient process for gathering diverse community opinions to inform an intervention Nancy Pandhi, Nora Jacobson, Neftali Serrano, Armando Hernandez, Elizabeth Zeidler- Schreiter, Natalie Wietfeldt, Zaher Karp A14: Using integrated administrative data to evaluate implementation of a behavioral health and trauma screening for children and youth in foster care Michael D. Pullmann, Barbara Lucenko, Bridget Pavelle, Jacqueline A. Uomoto, Andrea Negrete, Molly Cevasco, Suzanne E. U. Kerns A15: Intermediary organizations as a vehicle to promote efficiency and speed of implementation Robert P. Franks, Christopher Bory A16: Applying the Consolidated Framework for Implementation Research constructs directly to qualitative data: The power of implementation science in action Edward J. Miech, Teresa M. Damush A17: Efficient and effective scaling-up, screening, brief interventions, and referrals to treatment (SBIRT) training: a snowball implementation model Jason Satterfield, Derek Satre, Maria Wamsley, Patrick Yuan, Patricia O’Sullivan A18: Matching models of implementation to system needs and capacities: addressing the human factor Helen Best, Susan Velasquez A19: Agency characteristics that facilitate efficient and successful implementation efforts Miya Barnett, Lauren Brookman-Frazee, Jennifer Regan, Nicole Stadnick, Alison Hamilton, Anna Lau A20: Rapid assessment process: Application to the Prevention and Early Intervention transformation in Los Angeles County Jennifer Regan, Alison Hamilton, Nicole Stadnick, Miya Barnett, Anna Lau, Lauren Brookman-Frazee A21: The development of the Evidence-Based Practice-Concordant Care Assessment: An assessment tool to examine treatment strategies across practices Nicole Stadnick, Anna Lau, Miya Barnett, Jennifer Regan, Scott Roesch, Lauren Brookman-Frazee A22: Refining a compilation of discrete implementation strategies and determining their importance and feasibility Byron J. Powell, Thomas J. Waltz, Matthew J. Chinman, Laura Damschroder, Jeffrey L. Smith, Monica M. Matthieu, Enola K. Proctor, JoAnn E. Kirchner A23: Structuring complex recommendations: Methods and general findings Thomas J. Waltz, Byron J. Powell, Matthew J. Chinman, Laura J. Damschroder, Jeffrey L. Smith, Monica J. Matthieu, Enola K. Proctor, JoAnn E. Kirchner A24: Implementing prolonged exposure for post-traumatic stress disorder in the Department of Veterans Affairs: Expert recommendations from the Expert Recommendations for Implementing Change (ERIC) project Monica M. Matthieu, Craig S. Rosen, Thomas J. Waltz, Byron J. Powell, Matthew J. Chinman, Laura J. Damschroder, Jeffrey L. Smith, Enola K. Proctor, JoAnn E. Kirchner A25: When readiness is a luxury: Co-designing a risk assessment and quality assurance process with violence prevention frontline workers in Seattle, WA Sarah C. Walker, Asia S. Bishop, Mariko Lockhart A26: Implementation potential of structured recidivism risk assessments with justice- involved veterans: Qualitative perspectives from providers Allison L. Rodriguez, Luisa Manfredi, Andrea Nevedal, Joel Rosenthal, Daniel M. Blonigen A27: Developing empirically informed readiness measures for providers and agencies for the Family Check-Up using a mixed methods approach Anne M. Mauricio, Thomas D. Dishion, Jenna Rudo-Stern, Justin D. Smith A28: Pebbles, rocks, and boulders: The implementation of a school-based social engagement intervention for children with autism Jill Locke, Courtney Benjamin Wolk, Colleen Harker, Anne Olsen, Travis Shingledecker, Frances Barg, David Mandell, Rinad S. Beidas A29: Problem Solving Teletherapy (PST.Net): A stakeholder analysis examining the feasibility and acceptability of teletherapy in community based aging services Marissa C. Hansen, Maria P. Aranda, Isabel Torres-Vigil A30: A case of collaborative intervention design eventuating in behavior therapy sustainment and diffusion Bryan Hartzler A31: Implementation of suicide risk prevention in an integrated delivery system: Mental health specialty services Bradley Steinfeld, Tory Gildred, Zandrea Harlin, Fredric Shephard A32: Implementation team, checklist, evaluation, and feedback (ICED): A step-by-step approach to Dialectical Behavior Therapy program implementation Matthew S. Ditty, Andrea Doyle, John A. Bickel III, Katharine Cristaudo A33: The challenges in implementing muliple evidence-based practices in a community mental health setting Dan Fox, Sonia Combs A34: Using electronic health record technology to promote and support evidence-based practice assessment and treatment intervention David H. Lischner A35: Are existing frameworks adequate for measuring implementation outcomes? Results from a new simulation methodology Richard A. Van Dorn, Stephen J. Tueller, Jesse M. Hinde, Georgia T. Karuntzos A36: Taking global local: Evaluating training of Washington State clinicians in a modularized cogntive behavioral therapy approach designed for low-resource settings Maria Monroe-DeVita, Roselyn Peterson, Doyanne Darnell, Lucy Berliner, Shannon Dorsey, Laura K. Murray A37: Attitudes toward evidence-based practices across therapeutic orientations Yevgeny Botanov, Beverly Kikuta, Tianying Chen, Marivi Navarro-Haro, Anthony DuBose, Kathryn E. Korslund, Marsha M. Linehan A38: Predicting the use of an evidence-based intervention for autism in birth-to-three programs Colleen M. Harker, Elizabeth A. Karp, Sarah R. Edmunds, Lisa V. Ibañez, Wendy L. Stone A39: Supervision practices and improved fidelity across evidence-based practices: A literature review Mimi Choy-Brown A40: Beyond symptom tracking: clinician perceptions of a hybrid measurement feedback system for monitoring treatment fidelity and client progress Jack H. Andrews, Benjamin D. Johnides, Estee M. Hausman, Kristin M. Hawley A41: A guideline decision support tool: From creation to implementation Beth Prusaczyk, Alex Ramsey, Ana Baumann, Graham Colditz, Enola K. Proctor A42: Dabblers, bedazzlers, or total makeovers: Clinician modification of a common elements cognitive behavioral therapy approach Rosemary D. Meza, Shannon Dorsey, Shannon Wiltsey-Stirman, Georganna Sedlar, Leah Lucid A43: Characterization of context and its role in implementation: The impact of structure, infrastructure, and metastructure Caitlin Dorsey, Brigid Marriott, Nelson Zounlome, Cara Lewis A44: Effects of consultation method on implementation of cognitive processing therapy for post-traumatic stress disorder Cassidy A. Gutner, Candice M. Monson, Norman Shields, Marta Mastlej, Meredith SH Landy, Jeanine Lane, Shannon Wiltsey Stirman A45: Cross-validation of the Implementation Leadership Scale factor structure in child welfare service organizations Natalie K. Finn, Elisa M. Torres, Mark. G. Ehrhart, Gregory A. Aarons A46: Sustainability of integrated smoking cessation care in Veterans Affairs posttraumatic stress disorder clinics: A qualitative analysis of focus group data from learning collaborative participants Carol A. Malte, Aline Lott, Andrew J. Saxon A47: Key characteristics of effective mental health trainers: The creation of the Measure of Effective Attributes of Trainers (MEAT) Meredith Boyd, Kelli Scott, Cara C. Lewis A48: Coaching to improve teacher implementation of evidence-based practices (EBPs) Jennifer D. Pierce A49: Factors influencing the implementation of peer-led health promotion programs targeting seniors: A literature review Agathe Lorthios-Guilledroit, Lucie Richard, Johanne Filiatrault A50: Developing treatment fidelity rating systems for psychotherapy research: Recommendations and lessons learned Kevin Hallgren, Shirley Crotwell, Rosa Muñoz, Becky Gius, Benjamin Ladd, Barbara McCrady, Elizabeth Epstein A51: Rapid translation of alcohol prevention science John D. Clapp, Danielle E. Ruderman A52: Factors implicated in successful implementation: evidence to inform improved implementation from high and low-income countries Melanie Barwick, Raluca Barac, Stanley Zlotkin, Laila Salim, Marnie Davidson A53: Tracking implementation strategies prospectively: A practical approach Alicia C. Bunger, Byron J. Powell, Hillary A. Robertson A54: Trained but not implementing: the need for effective implementation planning tools Christopher Botsko A55: Evidence, context, and facilitation variables related to implementation of Dialectical Behavior Therapy: Qualitative results from a mixed methods inquiry in the Department of Veterans Affairs Sara J. Landes, Brandy N. Smith, Allison L. Rodriguez, Lindsay R. Trent, Monica M. Matthieu A56: Learning from implementation as usual in children’s mental health Byron J. Powell, Enola K. Proctor A57: Rates and predictors of implementation after Dialectical Behavior Therapy Intensive Training Melanie S. Harned, Marivi Navarro-Haro, Kathryn E. Korslund, Tianying Chen, Anthony DuBose, André Ivanoff, Marsha M. Linehan A58: Socio-contextual determinants of research evidence use in public-youth systems of care Antonio R. Garcia, Minseop Kim, Lawrence A. Palinkas, Lonnie Snowden, John Landsverk A59: Community resource mapping to integrate evidence-based depression treatment in primary care in Brazil: A pilot project Annika C. Sweetland, Maria Jose Fernandes, Edilson Santos, Cristiane Duarte, Afrânio Kritski, Noa Krawczyk, Caitlin Nelligan, Milton L. Wainberg A60: The use of concept mapping to efficiently identify determinants of implementation in the National Institute of Health--President’s Emergent Plan for AIDS Relief Prevention of Mother to Child HIV Transmission Implementation Science Alliance Gregory A. Aarons, David H. Sommerfeld, Benjamin Chi, Echezona Ezeanolue, Rachel Sturke, Lydia Kline, Laura Guay, George Siberry A61: Longitudinal remote consultation for implementing collaborative care for depression Ian M. Bennett, Rinad Beidas, Rachel Gold, Johnny Mao, Diane Powers, Mindy Vredevoogd, Jurgen Unutzer A62: Integrating a peer coach model to support program implementation and ensure long- term sustainability of the Incredible Years in community-based settings Jennifer Schroeder, Lane Volpe, Julie Steffen A63: Efficient sustainability: Existing community based supervisors as evidence-based treatment supports Shannon Dorsey, Michael D Pullmann, Suzanne E. U. Kerns, Nathaniel Jungbluth, Lucy Berliner, Kelly Thompson, Eliza Segell A64: Establishment of a national practice-based implementation network to accelerate adoption of evidence-based and best practices Pearl McGee-Vincent, Nancy Liu, Robyn Walser, Jennifer Runnals, R. Keith Shaw, Sara J. Landes, Craig Rosen, Janet Schmidt, Patrick Calhoun A65: Facilitation as a mechanism of implementation in a practice-based implementation network: Improving care in a Department of Veterans Affairs post-traumatic stress disorder outpatient clinic Ruth L. Varkovitzky, Sara J. Landes A66: The ACT SMART Toolkit: An implementation strategy for community-based organizations providing services to children with autism spectrum disorder Amy Drahota, Jonathan I. Martinez, Brigitte Brikho, Rosemary Meza, Aubyn C. Stahmer, Gregory A. Aarons A67: Supporting Policy In Health with Research: An intervention trial (SPIRIT) - protocol and early findings Anna Williamson A68: From evidence based practice initiatives to infrastructure: Lessons learned from a public behavioral health system’s efforts to promote evidence based practices Ronnie M. Rubin, Byron J. Powell, Matthew O. Hurford, Shawna L. Weaver, Rinad S. Beidas, David S. Mandell, Arthur C. Evans A69: Applying the policy ecology model to Philadelphia’s behavioral health transformation efforts Byron J. Powell, Rinad S. Beidas, Ronnie M. Rubin, Rebecca E. Stewart, Courtney Benjamin Wolk, Samantha L. Matlin, Shawna Weaver, Matthew O. Hurford, Arthur C. Evans, Trevor R. Hadley, David S. Mandell A70: A model for providing methodological expertise to advance dissemination and implementation of health discoveries in Clinical and Translational Science Award institutions Donald R. Gerke, Beth Prusaczyk, Ana Baumann, Ericka M. Lewis, Enola K. Proctor A71: Establishing a research agenda for the Triple P Implementation Framework Jenna McWilliam, Jacquie Brown, Michelle Tucker A72: Cheap and fast, but what is “best?”: Examining implementation outcomes across sites in a state-wide scaled-up evidence-based walking program, Walk With Ease Kathleen P Conte A73: Measurement feedback systems in mental health: Initial review of capabilities and characteristics Aaron R. Lyon, Meredith Boyd, Abigail Melvin, Cara C. Lewis, Freda Liu, Nathaniel Jungbluth A74: A qualitative investigation of case managers’ attitudes toward implementation of a measurement feedback system in a public mental health system for youth Amelia Kotte, Kaitlin A. Hill, Albert C. Mah, Priya A. Korathu-Larson, Janelle R. Au, Sonia Izmirian, Scott Keir, Brad J. Nakamura, Charmaine K. Higa-McMillan A75: Multiple pathways to sustainability: Using Qualitative Comparative Analysis to uncover the necessary and sufficient conditions for successful community-based implementation Brittany Rhoades Cooper, Angie Funaiole, Eleanor Dizon A76: Prescribers’ perspectives on opioids and benzodiazepines and medication alerts to reduce co-prescribing of these medications Eric J. Hawkins, Carol A. Malte, Hildi J. Hagedorn, Douglas Berger, Anissa Frank, Aline Lott, Carol E. Achtmeyer, Anthony J. Mariano, Andrew J. Saxon A77: Adaptation of Coordinated Anxiety Learning and Management for comorbid anxiety and substance use disorders: Delivery of evidence-based treatment for anxiety in addictions treatment centers Kate Wolitzky-Taylor, Richard Rawson, Richard Ries, Peter Roy-Byrne, Michelle Craske A78: Opportunities and challenges of measuring program implementation with online surveys Dena Simmons, Catalina Torrente, Lori Nathanson, Grace Carroll A79: Observational assessment of fidelity to a family-centered prevention program: Effectiveness and efficiency Justin D. Smith, Kimbree Brown, Karina Ramos, Nicole Thornton, Thomas J. Dishion, Elizabeth A. Stormshak, Daniel S. Shaw, Melvin N. Wilson A80: Strategies and challenges in housing first fidelity: A multistate qualitative analysis Mimi Choy-Brown, Emmy Tiderington, Bikki Tran Smith, Deborah K. Padgett A81: Procurement and contracting as an implementation strategy: Getting To Outcomes® contracting Ronnie M. Rubin, Marilyn L. Ray, Abraham Wandersman, Andrea Lamont, Gordon Hannah, Kassandra A. Alia, Matthew O. Hurford, Arthur C. Evans A82: Web-based feedback to aid successful implementation: The interactive Stages of Implementation Completion (SIC)TM tool Lisa Saldana, Holle Schaper, Mark Campbell, Patricia Chamberlain A83: Efficient methodologies for monitoring fidelity in routine implementation: Lessons from the Allentown Social Emotional Learning Initiative Valerie B. Shapiro, B.K. Elizabeth Kim, Jennifer L. Fleming, Paul A. LeBuffe A84: The Society for Implementation Research Collaboration (SIRC) implementation development workshop: Results from a new methodology for enhancing implementation science proposals Sara J. Landes, Cara C. Lewis, Allison L. Rodriguez, Brigid R. Marriott, Katherine Anne Comtois A85: An update on the Society for Implementation Research Collaboration (SIRC) Instrument Review Projec

Proceedings of the 3rd Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2015: advancing efficient methodologies through community partnerships and team science

It is well documented that the majority of adults, children and families in need of evidence-based behavioral health interventionsi do not receive them [1, 2] and that few robust empirically supported methods for implementing evidence-based practices (EBPs) exist. The Society for Implementation Research Collaboration (SIRC) represents a burgeoning effort to advance the innovation and rigor of implementation research and is uniquely focused on bringing together researchers and stakeholders committed to evaluating the implementation of complex evidence-based behavioral health interventions. Through its diverse activities and membership, SIRC aims to foster the promise of implementation research to better serve the behavioral health needs of the population by identifying rigorous, relevant, and efficient strategies that successfully transfer scientific evidence to clinical knowledge for use in real world settings [3]. SIRC began as a National Institute of Mental Health (NIMH)-funded conference series in 2010 (previously titled the “Seattle Implementation Research Conference”; $150,000 USD for 3 conferences in 2011, 2013, and 2015) with the recognition that there were multiple researchers and stakeholdersi working in parallel on innovative implementation science projects in behavioral health, but that formal channels for communicating and collaborating with one another were relatively unavailable. There was a significant need for a forum within which implementation researchers and stakeholders could learn from one another, refine approaches to science and practice, and develop an implementation research agenda using common measures, methods, and research principles to improve both the frequency and quality with which behavioral health treatment implementation is evaluated. SIRC’s membership growth is a testament to this identified need with more than 1000 members from 2011 to the present.ii SIRC’s primary objectives are to: (1) foster communication and collaboration across diverse groups, including implementation researchers, intermediariesi, as well as community stakeholders (SIRC uses the term “EBP champions” for these groups) – and to do so across multiple career levels (e.g., students, early career faculty, established investigators); and (2) enhance and disseminate rigorous measures and methodologies for implementing EBPs and evaluating EBP implementation efforts. These objectives are well aligned with Glasgow and colleagues’ [4] five core tenets deemed critical for advancing implementation science: collaboration, efficiency and speed, rigor and relevance, improved capacity, and cumulative knowledge. SIRC advances these objectives and tenets through in-person conferences, which bring together multidisciplinary implementation researchers and those implementing evidence-based behavioral health interventions in the community to share their work and create professional connections and collaborations

The external validity of randomized controlled trials: Outpatients and the psychotherapy outcome literature

In order to determine the extent to which published randomized controlled trials (RCTs) of psychotherapy can be generalized to a sample of community outpatients, a method of matching was employed, using information obtained from outpatient charts to inclusion and exclusion criteria from published RCT studies. The use of this procedure in study 1 revealed that 80% of the patients who had diagnoses represented in the RCT literature were judged eligible for at least one published RCT, but that 58% of the patients had primary diagnoses such as adjustment disorder or dysthymia, which were not represented in the existing psychotherapy outcome literature. The most common reasons that patients in this sample did not match with published RCTs for psychotherapy were (1) having a diagnosis that is not represented in the current RCT literature, (2) failing to meet minimum severity or duration criteria, and (3) failing to meet demographic criteria. Study 2 employed the same methodology with a sample of outpatients who were screened for, but not randomized into, an RCT. In this sample, more patients with substance use disorders or social anxiety disorder were not eligible for at least two RCTs. Common reasons that patients did not match with at least two published RCTs for psychotherapy included: (1) patients were in partial remission, (2) failure to meet minimum severity or duration criteria, (3) treatment with antidepressant medication, and (4) the disorder being studied was not primary (mostly for social anxiety patients). Study 3 involved comparing patients who were not randomized into studies for major depressive disorder to those who were randomized on diagnostic and demographic variables. Non-randomized patients were of a comparable level of severity as randomized patients, and tended to have either comparable or significantly lower rates of Axis I and II comorbidity, substance abuse, and recurrence. While more non-randomized patients were non-white and unemployed than randomized patients, these characteristics appeared to be associated with higher rates of comorbidity among the non-white, unemployed patients. The implications of these findings for future research and clinical practice are discussed

The external validity of randomized controlled trials: Outpatients and the psychotherapy outcome literature

In order to determine the extent to which published randomized controlled trials (RCTs) of psychotherapy can be generalized to a sample of community outpatients, a method of matching was employed, using information obtained from outpatient charts to inclusion and exclusion criteria from published RCT studies. The use of this procedure in study 1 revealed that 80% of the patients who had diagnoses represented in the RCT literature were judged eligible for at least one published RCT, but that 58% of the patients had primary diagnoses such as adjustment disorder or dysthymia, which were not represented in the existing psychotherapy outcome literature. The most common reasons that patients in this sample did not match with published RCTs for psychotherapy were (1) having a diagnosis that is not represented in the current RCT literature, (2) failing to meet minimum severity or duration criteria, and (3) failing to meet demographic criteria. Study 2 employed the same methodology with a sample of outpatients who were screened for, but not randomized into, an RCT. In this sample, more patients with substance use disorders or social anxiety disorder were not eligible for at least two RCTs. Common reasons that patients did not match with at least two published RCTs for psychotherapy included: (1) patients were in partial remission, (2) failure to meet minimum severity or duration criteria, (3) treatment with antidepressant medication, and (4) the disorder being studied was not primary (mostly for social anxiety patients). Study 3 involved comparing patients who were not randomized into studies for major depressive disorder to those who were randomized on diagnostic and demographic variables. Non-randomized patients were of a comparable level of severity as randomized patients, and tended to have either comparable or significantly lower rates of Axis I and II comorbidity, substance abuse, and recurrence. While more non-randomized patients were non-white and unemployed than randomized patients, these characteristics appeared to be associated with higher rates of comorbidity among the non-white, unemployed patients. The implications of these findings for future research and clinical practice are discussed

The Sustainability of Evidence-Based Interventions and Practices in Public Health and Health Care

There is strong interest in implementation science to address the gap between research and practice in public health. Research on the sustainability of evidence-based interventions has been growing rapidly. Sustainability has been defined as the continued use of program components at sufficient intensity for the sustained achievement of desirable program goals and population outcomes. This understudied area has been identified as one of the most significant translational research problems. Adding to this challenge is uncertainty regarding the extent to which intervention adaptation and evolution are necessary to address the needs of populations that differ from those in which interventions were originally tested or implemented. This review critically examines and discusses conceptual and methodological issues in studying sustainability, summarizes the multilevel factors that have been found to influence the sustainability of interventions in a range of public health and health care settings, and highlights key areas for future research